DDC  FILE_COP(j  ADA  08  640 


Ttchiical  Report  437 


LEVELr  ^ 


USEFULNESS  OF  READABILITY  FORMULAS  FOR 
ACHIEVING  ARMY  READABILITY  OBJECTIVES: 
RESEARCH  AND  STATE-OF-THE-ART  APPLIED 
TO  THE  ARMY’S  PROBLEM 


Richard  P.  Kern 


TRAINING  TECHNICAL  ARIA 


r  '  0* 

I 


U.  S.  Army 

Research  Institute  for  the  Behavioral  and  Social  Sciences 

January  1980 

Approved  for  public  releeee;  distribution  unlimited. 


80  7  10  044 


u.  S.  ARMY  RESEARCH  INSTITUTE 

FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 


A  Field  Operating  Agency  under  the  Jurisdiction  of  the 
Deputy  Chief  of  Staff  for  Personnel 


JOSEPH  ZEIDNER 
Technical  Director 


FRANKLIN  A.  HART 
Colonel,  US  Army 
Commander 


NOTICES 


DISTRIBUTION'  Primary  distribution  of  thu  report  has  been  made  by  ARI.  Pleaee  add  ran  correspondence 
concerning  dutnbution  of  reports  to:  U.  S.  Army  Research  Institute  for  the  Beftayloral  and  Social  Sciences. 
ATTN:  PERI-TP,  5001  Eisenhower  Avenue,  Alexandria,  Virginia  22333. 


PINAL  DISPOSITION:  This  report  may  be  destroyed  when  it  is  no  longer  needed.  Pleaee  do  not  return  it  to 
the  U.  S.  Army  Research  institute  for  the  Behavioral  and  Social  Sciences. 


MOTE:  The  findings  in  this  report  ere  not  to  be  construed  as  an  official  Department  of  the  Army  potition. 
unless  so  designated  by  other  authorised  documents. 


f 

Unclassified  _ 


SECURITY  CLASSIFICATION  of  THIS  PAGE  fWh#n  Data  Entered) 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

t.  REPORT  NUMBER  2.  GOVT  ACCESSION  NO. 

Technical  Report  437  ^ 

9.  RECIPIENT'S  CATALOG  NUMBER 

4.  TITLE  find  Submit) 

Usefulness  of  Readability  Formulas  for  Achieving 
Army  Readability  Objectives:  Research  id  State- 
of-the-Art  Applied  to  the  Army's  Problem 

5.  TYPE  OF  REPORT  A  PERIOO  COVERED 

6.  PERFORMING  ORG.  REPORT  NUMBER 

7.  author^; 

Richard  P.  Kern 

0.  CONTRACT  OR  GRANT  NUMBERf a) 

9.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

US  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences,  5001  Eisenhower  Avenue 
Alexandria,  VA  22333  p*'" 

10.  PROGRAM  ELEMENT.  PROJECT,  TASK 
AREA  A  WORK  UNIT  NUMBERS 

2Q162722A777 

II.  controlling  office  name  and  address 

US  Army  Administration  Center 

Ft  Benjamin  Harrison,  Ind.  46216 

12.  REPORT  date 

January  1980  / 

13.  NUMBER  OF  PAGES 

36 

U.  MONITORING  AGENCY  NAME  A  AODRESS(7 1  di fferent  from  Controlling  Office) 

15.  SECURITY  CLASS,  (of  thle  report) 

Unclassified 

1  S«.  DEC  L  ASSI F 1 C  A  Tl  ON  /  DO  WN  G  R  A  Dl  N  G 
SCHEDULE 

IS.  DISTRIBUTION  STATEMENT  (of  thia  Report) 

Approved  for  public  release,  distribution  unlimited. 

17.  DISTRIBUTION  STATEMENT  (ol  Ihm  tbtlrocl  ontorod  In  Block  30,  II  dlllortnl  from  Raporl) 

u.  supplementary  notes 

If.  KEY  WORDS  fCortfinu*  on  reeetee  aide  it  neceaaary  and  Identify  by  block  number) 

Readability  Standards 

Readability  Formulas 

Reading  Comprehension 

Reading  Skill  Levels 

Comprehension  of  Text 

This  report  was  prepared  to  assist  the  Army  in  evaluating  the  usefulness  of 
readability  formulas  in  identifying  material  that  will  be  comprehended  by 
readers  at  a  given  reading  skill  level. 

Existing  readability  methods  and  research  findings  were  reviewed  against 
three  major  criterion  questions.  These  questions  and  conclusions  reached 
for  each  are  as  follows: _ 

y  /Ti.»  *  v 

00  ,  1473  ioition  of  i  hov  •»  is  obsolete  Unclassified 


i  SECURITY  CLASSIFICATION  OF  THIS  PACE  fWi»n  Data  Brtlotod) 


Unclassified 


SECURITY  CLASSIFICATION  OF  THU  PAOE<TW<<H  Dim  EnlwQ 


(0  Question:  How  effectively  do  formulas  predict  comprehension  at 
targeted  reading  grade  levels? 

Conclusion:  Formulas  cannot  match  material  to  reader. 

(a)  Question:  Does  rewriting  to  lower  readability  scores  increase  compre¬ 
hension? 

Conclusion:  Rewriting  to  lower  the  formula  reading  grade  level  score 
does  not  increase  comprehension. 

(  3)  Question:  How  will  formula  score  requirements  affect  production  of 
written  materials? 

Conclusion:  Requiring  that  text  be  written  to  satisfy  a  targeted 
reading  grade  level  score  focuses  attention  on  meeting  the  score  requirement 
rather  than  on  organizing  the  material  to  meet  the  readers*  information  needs. 

An  approach  is  outlined  for  developing  a  broader  based  method  and  criteria  to 
by-pass  the  serious  deficiencies  of  present  day  readability  formulas. 

•v 


ii  Unclassified 

SECURITY  CLASSIFICATION  OF  THIS  PAOEf*ft««  Dim  Mnlmi 


USEFULNESS  OF  READABILITY  FORMULAS  FOR 
\  ACHIEVING  ARMY  fEADABILITY  OBJECTIVES: 

\  RESEARCH  AND  STATE-OF-THE-ART  APPLIED 

\ 

TO  THE  ARMY’S  PROBLEM. 

Richard  P.jKern  J 
Joyce  L.  Shields,  Team  Chief 


Submitted  by: 

Milton  S.  Katz,  Chief 
TRAINING  TECHNICAL  AREA 


Approved  by: 


E.  Ralph  Dusek,  Director 
PERSONNEL  AND  TRAINING 
RESEARCH  LABORATORY 


U.S.  ARMY  RESEARCH  INSTITUTE  FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 
5001  Eisenhower  Avenue.  Alexandria,  Virginia  22333 


Office,  Deputy  Chief  of  Staff  for  Personnel 
Department  of  the  Army 


/ 


ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&O  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


iv 


FOREWORD 


The  Training  Technical  Area  of  che  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI)  conducts  a  program  of  research  in 
support  of  the  systems  engineering  of  training.  A  major  part  of  this 
research  is  to  develop  the  fundamental  data  and  technology  necessary  to 
put  into  the  field  integrated  systems  for  improving  individual  job 
performance.  Such  systems  include  Skill  Qualification  Testing  (SQT) , 
job  performance  aids,  training  courses  in  schools  and  in  the  field, 
performance  criteria,  and  management  and  feedback  systems. 

This  r  ort  was  prepared  in  response  to  a  request  for  Technical 
Advisory  Service  from  Director  of  Evaluation,  US  Army  Administration 
Center,  Fort  Benjamin  Harrison,  Indiana.  This  request  was  for  technical 
assistance  in  evaluating  and  identifying  methods  that  will  assist  the 
Army  in  achieving  objectives  for  improved  readability  of  Army  publications. 

This  review  focuses  on  a  specific  methodological  problem  basic  to 
job  aid  research  currently  being  conducted  under  Army  Project  2Q162722A777, 
Training  Technology. 


Technical  Director 
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USEFULNESS  OF  READABILITY  FORMULAS  FOR  ACHIEVING  ARMY  READABILITY 
OBJECTIVES:  Research  and  State-of-the-Art  Applied  to  the  Army’s  Problem 


BRIEF 


Requirement : 

Evaluate  existing  methods  for  determining  readability  of  written 
materials,  and  determine  the  most  effective  method  for  achieving  Army 
readab il ity  ob j ec  t ives . 

Procedure : 

Existing  readability  methods  were  reviewed  against  three  major 
criterion  questions  presented  below  with  conclusions.  This  review  was 
based  on  the  existing  research  literature.  Data  were  obtained  from 
research  reports  on  the  development  and  revision  of  a  number  of  readability 
formulas.  These  data  were  used  to  examine  the  extent  of  agreement 
between  readers'  comprehension  test  scores  and  the  reading  difficulty 
level  score  obtained  by  applying  readability  formulas  to  the  same  passages. 
In  addition,  research  was  reviewed  on  the  effects  of  rewriting  text  to 
improve  ease  of  comprehension. 

Findings : 

Question:  How  effectively  do  formulas  predict  comprehension  at 
targeted  reading  grade  levels? 

Conclusion:  Formulas  cannot  match  material  to  reader 

Current  readability  formulas  provide  unreliable  predictions  of 
reading  skill  levels  at  which  text  can  be  comprehended  and  are  therefore 
unsuitable  for  meeting  Army  readability  objectives. 

Example:  Text  comprehended  by  readers  with  7th  to  8th  grade  reading 
skills  may  yield  formula  readability  scores  ranging  from  the  3rd  to  11th 
grade  levels.  Text  understood  by  readers  with  9th  to  10th  grade  reading 
skills  may  yield  formula  scores  ranging  from  the  6th  to  the  12th  reading 
grade  levels. 

Current  readability  formulas  appear  most  reliable  when  used  to  make 
a  gross  screening  distinction  between  material  that  is  above  or  below 
about  the  10th  grade  difficulty  level.  Soldiers  who  read  below  the  10th 
grade  skill  level  will  generally  not  be  able  to  readily  comprehend  text 
that  scores  above  the  10th  grade  difficulty  level. 
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Question:  Does  rewriting  to  lower  readability  scores  Increase 
comprehension? 

Conclusion:  Rewriting  to  lower  the  formula's  reading  grade  level 
score  does  not  increase  comprehension. 

Research  reviewed  indicates  that  rewriting  text  to  reduce  its  reading 
difficulty  level  (formula  score)  by  using  shorter  words  and  sentences  to 
"say  the  same  thing"  does  not  result  in  corresponding  Increases  in 
readers’  comprehension. 

Question:  How  will  formula  score  requirements  affect  production  of 
written  materials? 

Conclusion:  Requiring  that  text  be  written  to  satisfy  a  targeted 

reading  grade  level  score  focuses  attention  on  meeting 
the  score  requirement  rather  than  on  organizing  the 
material  to  meet  the  readers'  information  needs]! 

Criteria  used  to  identify  writing  that  meets  readability  standards 
should  be  criteria  that  writers  and  editors  can  be  encouraged  to  use  as 
a  guide  to  detect  problems  and  improve  their  text.  Present  day  read¬ 
ability  formulas  cannot  provide  this  guidance.  By  shifting  attention 
from  content  features  to  formula  scores,  readability  standards  my  even 
promote  production  of  writing  that  is  harder  to  understand. 

Utilization: 

This  report  presents  an  approach  for  developing  a  broad-based  method 
for  establishing  readability  standards  and  for  evaluating  proposed 
publications  against  these  standards.  This  method  will  address  the 
following  objectives  readability  formulas  cannot  address: 

(1)  Provide  a  method  and  criteria  writers  and  editors  can  use  to 
develop  and  Improve  their  material. 

(2)  Identify  specific  kinds  of  problems  present  when  material  does 
not  meet  standards. 

(3)  Apply  to  illustrations  and  all  other  parts  of  a  publication. 
(Readability  formulas  can  only  be  applied  to  passages  of  written  text) . 
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USEFULNESS  OF  READABILITY  FORMULAS  FOR  ACHIEVING  ARMY  READABILITY 
OBJECTIVES:  RESEARCH  AND  STATE-OF-THE-ART  APPLIED  TO  THE  ARMY'S  PROBLEM 


BACKGROUND 

Department  of  the  Army  has  undertaken  a  long-range  program  to  improve 
the  Army  publication  system.  A  major  objective  of  this  program,  improving 
the  readability  of  Army  publications,  is  addressed  by  Department  of  the 
Army  Circular  310-9  (15  December  1978).  This  circular  prescribes  use  of 
readability  formulas  as  the  major  method  for  determining  when  readability 
objectives  have  been  met  for  a  proposed  publication.  Commanding  General, 

US  Army  Administration  Center  (ADMINCEN) ,  and  The  Adjutant  General  are 
tasked  in  this  circular  to  review  existing  methods  for  measuring  readability 
and  to  develop  a  single  method  for  use  in  preparing  Army  publications. 

The  present  paper  was  prepared  in  response  to  the  Administration  Center's 
request  for  technical  assistance  in  formulating  their  recommendations. 

OBJECTIVES  OF  THIS  PAPER 

1.  Evaluate  the  usefulness  of  readability  formulas  and  other  methods 
for  achieving  Army  readability  objectives. 

2.  Determine  the  method  which  most  effectively  assures  achievement 
•  of  Army  readability  objectives. 

THE  ARMY'S  PROBLEM 

The  Army's  objective  is  to  produce  publications  which  all  of  its 
reader  population  can  use  effectively.  To  achieve  this  objective  publications 
must  satisfy  two  interrelated  conditions.  They  must:  (a)  clearly 
address  the  job  and  training  needs  of  the  target  audience,  and  (b)  be 
written  in  a  style  that  is  easily  comprehended  by  soldiers  in  the  target 
audience.  The  Army  has  made  considerable  progress  in  applying  Instructional 
Systems  Development  (ISD)  as  a  tool  to  Identify  content  oriented  to  the 
Job  and  training  needs  of  a  target  audience.  Present  need  is  for  a 
method  to  use  in  monitoring  and  improving  the  production  of  written 
materials  to  assure  that  readers  in  the  target  audience  will  be  able  to 
comprehend  them.  Of  particular  concern  are  the  below  average  to  average 
readers.  This  concern  is  reflected  in  the  reading  difficulty  level 
standards  specified  in  the  recent  Department  of  the  Array  Circular  310-9 
(Improving  the  Readability  of  Army  Publications).  For  example,  these 
standards  set  a  target  reading  difficulty  level  of  the  7th  grade  for 
training  and  job  publications  prepared  for  soldiers  in  pay  grades  E-l 
through  E~4.  Data  recently  reported  by  Mathews,  Valentine  and  Sellman 
suggest  that  as  many  as  30  percent  of  the  soldiers  entering  the  Army 
score  below  the  7th  Reading  Grade  Level  (RGL)  on  a  Gates-MacGinitie 
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Reading  Test  (Mathews  et  al.,  1978).  These  data  suggest  good  reason  for 
the  Army's  concern  over  providing  job  and  training  publications  that  can 
be  used  effectively  by  the  below  average  reader.  The  question  is,  can 
readability  formulas  provide  a  sound  basis  for  achieving  the  Army's 
readability  objectives?  Are  there  alternative  approaches  that  will  be 
more  effective? 

WHAT  DOES  A  READABILITY  FORMULA  PREDICT? 

Readability  formulas  provide  a  method  for  rank  ordering  difficulty 
level  of  textual  materials  in  relation  to  average  comprehension  test 
performance  of  students  at  successive  school  grade  levels.  These  rank 
order  scores  are  usually  expressed  as  reading  grade  levels  (RGL) . 

School  grade  levels  provide  a  means  of  indexing  general  growth  in 
language  skills  displayed  by  the  average  student  over  the  course  of 
their  exposure  to  school  and  their  culture.  A  great  deal  of  their 
appeal  rests  on  the  assumption  that  if  a  relatively  difficult  passage 
can  be  pegged,  say,  at  the  12th  school  grade  level,  then  the  user  knows 
how  "high"  that  is  compared  to  one  pegged  to  some  lower  grade  level, 
say,  the  5th  grade  level.  To  accomplish  this  "pegging,"  readability 
formulas  are  developed  to  predict  the  school  grade  level  at  which  students, 
if  given  a  comprehension  teat  on  the  passage  in  question,  will  achieve 
an  average  test  score  equal  to  some  pre-aet  criterion,  for  example,  70 
percent  correct.  For  Instance,  a  10th  grade  level  readability  score 
represents  a  prediction  that  approximately  50  percent  of  the  10th  grade 
students  will  be  able  to  read  the  passage  and  pass  a  comprehension  test 
given  Immediately  after  the  reading.  If  this  prediction  could  be  made 
with  complete  accuracy  we  would  expect  less  than  50  percent  of  the  9th 
grade  students  and  more  than  50  percent  of  the  11th  grade  students  to 
pass  the  comprehension  test.  Obviously  these  predictions  cannot  be  made 
with  complete  accuracy.  So  the  question  is  how  much  error  can  be  expected 
in  these  predictions?  What  are  the  implications  of  the  amount  of  error 
and  sources  of  error  for  the  way  the  Army  proposes  to  set  standards  and 
use  these  formulas? 

USEFULNESS  OF  READABILITY  FORMULAS  FOR  ACHIEVING  ARMY  OBJECTIVES 

Usefulness  of  readability  formulas  for  achieving  Army  objectives 
hinges  on  three  major  considerations:  (a)  How  effectively  do  formulas 
predict  comprehension  at  targeted  reading  grade  levels?  (b)  Does 
rewriting  to  lower  readability  scores  result  in  Increased  comprehension? 
And,  (c)  How  will  formula  score  requirements  affect  production  of  written 
materials? 


How  Effectively  Do  Formulas  Predict  Comprehension  At  Targeted  Reading 
Skill  Levels? 

Data  contained  in  two  recent  studies  permit  a  look  at  how  well 
readability  formulas  predict  the  grade  level  criteria  used  in  the 
development  of  the  formulas  (Caylor  et  al.,  1973;  Kincaid  et  al.,  1975). 


Both  of  these  studies  were  concerned  with  development  of  readability 
formulas  for  military  personnel  using  military  job  and  training  materials. 
Cay lor  et  al.  developed  the  FORCAST  formula  using  Army  recruits  and 
passages  from  Army  job  and  training  publications.  Kincaid  et  al. 
recalculated  six  readability  formulas  using  Navy  enlisted  personnel  with 
service  of  six  months  or  less  and  passages  selected  from  Navy  job  and 
training  material. 

Comprehension  Criteria  Expressed  As  Grade  Levels.  Caylor  and  Kincaid 
used  the  same  method  for  establishing  the  grade  level-comprehension 
criteria  to  be  predicted,  Jhis  method  consisted  of  first  administering 
a  standardized  reading  test  to  the  people  in  the  sample  to  identify 
their  reading  grade  level  placement.  The  experimental  passages  were 
then  given  in  the  form  of  cloze  tests.  Grade  level  comprehension 
criteria  were  then  determined  by  identifying  for  each  passage  the  lowest 
reading  grade  level  in  which  50  percent  of  the  men  reading  at  that  grade 
level  achieved  a  cloze  score  at  or  above  the  35  percent  correct  criterion 
level.  The  resulting  criterion  reading  grade  levels,  called  scaled 
Reading  Grade  Levels  (RGL) ,  are  shown  in  Table  1  for  the  18  passages 
used  by  Kincaid  and  in  Table  2  for  the  12  passages  used  in  developing 
the  FORCAST.  The  formulas  were  then  developed,  using  multiple  correlational 
techniques,  to  predict  the  scaled  RGL  value  for  each  passage.  The 
FORCAST  formula  uses  number  of  one-syllable  words  in  a  150  word  passage 
as  its  predictor.  The  recalculated  Flesch  formula  uses  average  number 
of  syllables  per  word  and  average  number  of  words  per  sentence  to  predict 
the  scaled  RGL. 


^Different  standardized  reading  tests  were  used  in  the  two  studies.  Caylor 
et  al.  used  the  United  States  Armed  Forces  Institute  (USAFI)  Achievement 
Tests  III,  Abbreviated  Edition,  Form  A  (a  special  printing  of  the  Metropolitan 
Achievement  Test ,  Advanced  Battery) .  Reading  Grade  Level  scores  on  this  test 
can  range  from  1.3  to  12.9.  Kincaid  et  al.  used  two  forms  of  the  Gates- 
MacGinitle  Reading  Test.  Form  E  (grade  levels  7-9)  was  given  to  personnel 
with  GCT  scores  below  55.  Form  F  (grade  levels  10-12)  was  given  to  those  with 
GCT  scores  of  55  and  above. 
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Cloze  tests  systematically  omit  words  which  testees  are  asked  to  fill  in  as 
a  measure  of  comprehension.  Caylor  et  al.  used  five  cloze  tests  for  each  of 
their  12  passages.  Version  1  deleted  every  fifth  word  counting  from  the  first 
word  in  the  passage.  Version  2  deleted  every  fifth  word  counting  from  the  s 
second  word  in  the  passage  and  so  forth.  Kincaid  et  al.  apparently  used 
only  one  version  for  each  passage. 

3 

Based  on  earlier  research  35  percent  correct  on  a  cloze  test  is  assumed  to 
be  equivalent  to  70  percent  correct  on  a  multiple  choice  comprehension  test. 
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TABLE  1 


Comparison  of  Scaled  Reading  Grade 

Level  (RGL) 

Values  with  Recalculated 

Formula 

Scores  for 

the  Eighteen  Passages  used  by  Kincaid 

et  al. ,  1975. 

Recalculated  Formulas 

Passage 

Scaled  Kincaid- 

F0RCAST 

ARI 

FOG 

Gunning 

Number 

RGL 

Flesch 

(Rev. ) 

(Rev.) 

Count  (Rev.) 

FOG  (Rev.) 

15 

5.5 

8.4 

7.9 

7.7 

10.1 

9.2 

17 

5.8 

9.3 

11.8 

10.2 

9.9 

11.0 

1 

6.9 

9.7 

8.0 

8.5 

10.9 

7.7 

5 

7.0 

7.1 

9.7 

8.6 

7.5 

6.9 

9 

7.8 

8.0 

8.1 

8.0 

8.9 

5.9 

10 

8.9 

11.7 

10.5 

11.9 

12.9 

12.0 

13 

8.9 

10.0 

9.0 

11.5 

11.0 

8.5 

11 

9.0 

8.1 

10.6 

7.0 

6.5 

7.3 

18 

9.8 

6.6 

8.0 

10.6 

6.7 

7.4 

14 

10.8 

12.5 

12.1 

10.6 

12.2 

12.0 

4 

10.9 

8.2 

9.5 

7.9 

7.3 

7.5 

16 

11.8 

13.8 

13.9 

12.7 

12.5 

15.4 

7 

12.0 

11.7 

11.8 

11.3 

11.6 

10.6 

3 

16.0 

12.7 

13.0 

12.4 

11.4 

12.7 

6 

16.0 

12.3 

13.8 

11.3 

11.2 

10.8 

2 

16.1 

16.7 

15.6 

18.3 

14.4 

12.1 

12 

16.1 

11.8 

8.7 

12.6 

13.5 

9.7 

8 

16.3 

14.7 

13.5 

14.7 

16.4 

14.9 

Note: 

Scaled  RGL 

and  Recalculated 

formula  values  for  the 

Flesch,  Automated 

Readability  Index 

(ARI)  and  the  FOG  count  are  : 

from  Table 

2,  Kincaid  et  al 

•  f 

1975. 

Values  for 

the  Recalculated  Gunning  FOG 

Index  and 

revised  FORCAST 

were  calculated  by  John  Brand,  US  Army  Administration  Center  (USAADMINCEN) 
and  the  author  based  on  data  provided  by  Kincaid. 
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TABLE  2 


Comparison  of  Scaled  Reading  Grade  Level  (RGL)  Values  with  Formula 
Scores  for  the  Twelve  Passages  used  in  the  Development  of  the  FORCAST 
Formula  by  Cay lor  et  al.,  1973. 

Formula  RGL  Values 


Passage 

Scaled 

FORCAST 

Dale- 

F-J-P 

Number 

RGL 

(Original) 

Chall 

Flesch 

21 

6 

8.6 

7-8 

7 

12 

7 

7.8 

7-8 

6 

11 

7 

7.6 

5-6 

5 

13 

7.3 

9.4 

7-8 

8-9 

23 

9.1 

10.1 

11-12 

13-16 

22 

9.6 

10.7 

9-10 

10-12 

15 

11.4 

12.1 

13-15 

13-16 

16 

11.8 

13.2 

16+ 

16+ 

26 

12.0 

12.2 

16+ 

16+ 

25 

12.0 

13.2 

16+ 

16+ 

24 

12.1 

11.3 

13-15 

13-16 

14 

13.0 

10.9 

13-15 

13-16 

Note :  All 

values  were  taken 

from  Cay lor  et 

al.,  1973. 

aF-J-P  Flesch  is  the  Farr-Jenkins-Paterson  revision  of  the  Flesch 
Reading  Ease  formula. 
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Comparison  of  Predicted  RGL  With  Scaled  RGL  Criteria.  Tables  1  and 
2  compare  the  scaled  RGL  values  for  the  passages  in  each  study  with  the 
RGL  score  obtained  by  applying  the  formula  to  the  passage.  The  purpose 
of  these  comparisons  is  to  Illustrate  the  fact  that  present  readability 
formulas  do  not  predict  comprehensibility  of  material  for  specific 
reading  grade  skill  levels.  They  simply  do  not  possess  that  precision. 

In  fact,  inspection  of  the  comparisons  shown  in  Table  1  for  the  Kincaid- 
Flesch  suggests  that  the  precision  of  this  formula  is  limited  to  predicting 
whether  comprehensibility  of  the  material  (Scaled  RGL)  is  above  or  below 
about  the  tenth  RGL. 

Range  Of  Formula  Scores  For  Scaled  RGL  Intervals.  Lack  of  precision 
in  these  predictions  is  illustrated  by  comparing  the  range  of  scores 
obtained  from  a  given  formula  with  the  scaled  RGL  scores  for  the  same 
passages.  The  scaled  RGL  criteria  and  formula  scores  for  the  passages 
are  shown  in  Table  3  for  the  Klncaid-Flesch.  Suppose  that  the  Army 
wanted  material  that  could  be  readily  understood  by  soldiers  reading  at 
the  5th  to  6th  RGL.  The  first  row  in  Table  3  shows  that  the  passages 
that  met  grade  level  comprehension  criteria  at  the  5th  to  6th  RGL  could 
have  Kincald-Flesch  scores  ranging  from  the  8th  to  the  9th  grade  level. 
Likewise  soldiers  whose  reading  skills  were  at  the  7th  to  8th  grade 
level  could  receive  material  ranging  in  Kincaid-Flesch  scores  from  the 
7th  to  the  11th  RGL.  The  range  of  Kincaid-Flesch  scores  for  passages 
that  required  reading  skills  at  the  9th  to  10th  grade  level  is  even 
broader,  ranging  from  the  6th  to  the  12th  grade  readability  score.  The 
right  hand  column  of  Table  3  shows  corresponding  data  used  in  developing 
the  FORCAST  formula.  It  should  be  pointed  out  that  the  lack  of  precision 
in  predicting  specific  reading  grade  skill  levels  just  illustrated  is 
not  specific  to  the  Kincaid-Flesch.  A  similar  lack  of  precision  is 
characteristic  of  all  of  the  formulas  in  Tables  1  and  2. 

Range  Of  Formula  Scores  When  Applied  To  New  Passages.  What  happens 
when  formulas  are  Sapplied  to  new  passages  not  used  in  developing  the 
formula?  Tables  presented  above  have  illustrated  the  lack  of  precision 
in  predicting  grade  level  comprehension  criteria  for  passages  used  in 
developing  the  formulas.  These  are  the  conditions  when  the  formula 
should  give  its  most  accurate  prediction.  We  would  expect  more  error,  in 
prediction  when  the  formula  is  applied  to  new  passages.  The  author 
calculated  the  FORCAST  scores  for  the  18  passages  Kincaid  et  al.  used  in 
their  study.  Kincwid-Flesch  scores  were  also  calculated  for  the  12 
passages  used  in  the  development  of  the  FORCAST  formula.  Table  4 
compares  the  results  obtained  when  each  formula  was  applied  to  its  own 
developmental  passages  with  the  results  obtained  when  each  formula  is 
applied  to  the  other  formula's  developmental  passages.  The  last  column 
in  Table  4  for  each  formula  represents  an  estimate,  based  on  these 
available  data,  of  the  range  in  formula  scores  we  might  expect  for 
material  meeting  comprehension  criteria  at  each  of  the  scaled  grade 
level  intervals.  These  data  should  severely  shake  the  confidence  of 
those  advocating  production  standards  that  will  match  RGL  of  materials 
with  RGL  of  readers. 
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TABLE  3 


Comparison  of  Scaled  Reading  Grade  Level  (RGL)  Values  with  Range  of 
Formula  Values  Obtained  on  Passages  used  In  Development  of  the  Kincald- 
Flesch  and  FORCAST  Formulas. 

Scaled  RGL 
Interval 

Klncaid-Fleach 
Kincaid  Passages 

FORCAST 

FORCAST  Passages 

5-6 

8-9 

8 

7-8 

7-11 

7-9 

9-10 

6-12 

10 

11-12 

11-13 

11-13 

13+ 


11-16 


10 


TABLE  4 


Range  of  Formula  Values  Obtained  by  Applying  Klncald-Flesch  and  FORCAST 
Formulas  to  Passages  used  In  Development  of  Formulas  and  Mew  Passages. 

Klncald-Flesch  Formula 


Scaled  RGL 

Kincaid 

New  (FORCAST) 

Composite 

Interval 

Passages 

Passages 

Estimate  of  Range 

5-6 

8-9 

5 

5-9 

7-8 

7-11 

3-6 

3-11 

9-10 

6-12 

7-12 

6-12 

11-12 

11-13 

14-21 

11-21 

13+ 

11-16 

11 

11-16 

FORCAST  Formula 


Scaled  RGL 

FORCAST 

New  (Kincaid) 

Composite 

Interval 

Passages 

Passages 

Estimate  of  Range 

5-6 

8 

9-11 

8-11 

7-8 

7-9 

9-10 

7-10 

9-10 

10 

9-11 

9-11 

11-12 

11-13 

11-12 

11-13 

13+ 

10 

10-13 

10-13 

Conclusion:  Formulas  Cannot  Match  Material  To  Reader.  Readability 
formulas  do  not  provide  reliable  predictions  of  reading  skill  levels 
(RGL)  at  which  material  can  be  adequately  comprehended.  This  conclusion 
is  not  restricted  to  the  Kincaid-Flesch  and  the  FORCAST.  All  of  the 
readability  formulas  shown  in  Tables  1  and  2  have  high  rank  order  agreement 
with  each  other. 

Does  Rewriting  To  Lower  Readability  Scores  Result  In  Increased  Comprehension? 

The  wide  range  of  error  possible  in  using  a  readability  score  to 
predict  whether  or  not  material  will  be  appropriate  for  soldiers  at  the 
lower  reading  skill  levels  was  illustrated  above  in  Tables  1-4.  These 
tables  demonstrate  quite  clearly  that  present  day  readability  formulas 
can  not  be  used  to  match  difficulty  level  of  the  material  to  reading 
skill  levels  of  below  average  readers. 

People  concerned  with  producing  understandable  reading  material  for 
below  average  adult  readers  advance  a  second  argument  for  use  of  readability 
standards.  This  argument  is  that  lowering  the  readability  levels  of  the 
material  will  result  in  higher  levels  of  comprehension  by  the  below 
average  reader.  This  apparently  is  the  assumption  on  which  readability 
standards  in  Department  of  the  Army  Circular  310-9  are  based. 

Klare's  Review  of  Research  Literature.  Klare  reviewed  36  experimental 
studies  performed  to  study  the  effects  of  lowering  or  raising  the  readability 
score  level  of  passages  on  comprehension  of  the  material  (Klare,  1976). 

The  general  approach  in  this  type  of  research  is  to  rewrite  a  given 
passage  to  make  it  easier  (or  harder)  than  the  original  passage  without 
changing  content  meaning.  "Easier"  or  "harder"  is  determined  by  applying 
a  readability  formula.  The  easier  and  harder  versions  of  the  same 
passage  are  then  given  to  different  groups  of  readers  and  comprehension 
scores  are  obtained  and  compared.  Klare  concluded  that  differences  in 
comprehension  were  obtained  in  19  of  the  36  studies,  no  differences  were 
found  in  11  studies,  and  the  remaining  6  studies  produced  "mixed"  results. 

The  importance  of  Klare's  review  to  the  present  concern  is  that  it 
suggests  that  present  day  optimism  that  simply  reducing  reading  difficulty 
level  of  material  will  Increase  the  reader's  comprehension  stands  on 
shaky  ground.  Recent  research  examining  this  same  question  has  been 
carried  out  by  the  Navy  and  by  the  Air  Force. 

Recent  Air  Force  Research  On  Lowering  Readability  Level  Of  Text. 

The  Air  Force  studied  basic  trainees  at  two  reading  test  skill  levels, 

8th  and  10th  Reading  Grade  Level  (Kniffin  et  al.,  1979).  The  purpose  of 
this  research  was  to  measure  the  effects  upon  reading  comprehension  of 
three  sizes  of  literacy  gap,  zero  gap  (readability  score  numerically 
equal  to  tested  RGL),  -2  gap  (readability  score  2  RGL  higher  than  tested 
RGL) ,  and  -4  gap  (readability  score  4  RGL  higher  than  tested  RGL) .  Four 
versions  of  two  different  passages  selected  from  Air  Force  training 
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TABLE  5 


Average  Comp rehens ion  Scores  (Percent  Correct)  Obtained  by  Reading 
Skill  Levels  at  each  Reading  Difficulty  Level  Version  of  Two  Passages 
(Kniffin  et  al,  1979). 


Reading  Difficulty  Level  of  Material 


Reading  Grade  Level 

of  Airmen 

8th  GL 

10th  GL 

12th  GL 

14th  GL 

10th  RGL 

Passage  1 

X 

67 

64 

62 

Passage  2 

X 

81 

78 

79 

8th  RGL 

Passage  1 

58 

61 

56 

X 

Passage  2 

82 

76 

73 

X 

x  x 


materials  were  prepared.  The  10th  reading  grade  level  subjects  received 
versions  of  the  two  passages  written  at  the  10th,  12th,  and  14th  grade 
readability  level  (Kincaid-Flesch  formula) .  The  8th  reading  grade  level 
subjects  received  versions  written  at  the  8th,  10th  and  12th  grade 
readability  levels.  The  same  52- item  multiple-choice  comprehension  test 
for  each  passage  was  used  with  all  subjects.  The  authors  conclude  that 
the  literacy  gap  produced  a  statistically  significant  but  small  effect 
upon  comprehension  scores. 

Table  5  presents  the  average  comprehension  scores  (percent  correct) 
obtained  by  each  group  in  the  Air  Force  study.  These  scores  illustrate 
again  the  inadequacy  of  readability  formulas  for  predicting  grade  level 
comprehension  criteria.  Note  that  readers  at  the  8th  grade  level  do  not 
reach  the  predicted  70  percent  comprehension  score  for  passage  1  on  the 
version  written  at  the  8th  grade  difficulty  level.  In  addition  they 
appear  to  do  almost  as  well  on  the  12th  grade  version  of  passage  1  as 
they  did  on  the  8th  grade  version.  Also  note  the  differences  in  comprehension 
scores  for  passages  1  and  2  at  the  same  reader  skill  level  when  the  two 
passages  each  received  the  same  readability  score.  Passage  2  is  consistently 
easier. 

Recent  Navy  Research  On  Lowering  Readability  Of  Text.  A  report  on 
the  Navy* s  research  is  not  yet  available.  However,  major  features  of 
the  research  plan  and  main  conclusions  were  given  in  an  oral  presentation 
by  Duffy  (Duffy,  1979).  Navy  research  studied  the  effects  of  careful 
rewriting  "to  formula"  on  comprehension  at  different  reading  skill 
levels . 

Navy  researchers  took  the  eight  passages  used  in  the  Nelson-Denny 
Reading  Test  and,  treating  them  as  a  set,  prepared  three  different 
versions  of  the  set.  One  version  was  prepared  by  changing  only  vocabulary. 
Shorter  word  synonyms  taken  from  standard  4th  grade  vocabulary  lists 
were  substituted  for  approximately  25  percent  of  the  passage  words. 

This  resulted  in  lowering  the  average  Kincaid-Flesch  score  from  11.5  to 
10.1  Reading  Grade  Le.vel  (RGL) .  A  second  version  was  prepared  by  only 
shortening  sentences.  This  resulted  in  an  average  Kincaid-Flesch  score 
of  7.3  RGL.  A  third  version  Involved  both  vocabulary  and  sentence 
changes  and  dropped  the  average  Kincaid-Flesch  score  to  5.5  RGL. 

The  Navy  conducted  four  different  studies  with  the  original  and  the 
three  revised  versions  of  these  passages.  In  each  study  the  four  versions 
were  presented  to  sailors  at  high  (10.5  and  above),  middle  (8.5  to  10.5) 
and  low  (7  to  8.5)  RGL.  In  Study  1  the  standard  Nelson-Denny  time 
limits  were  imposed  and  questions  and  text  were  presented  together  as  in 
the  usual  reading  test  procedure.  Easier  versions  did  not  produce 
improved  comprehension  at  any  of  the  three  reading  skill  levels.  In 
Study  2  time  limits  were  removed  and  questions  were  previewed  prior  to 
seeing  the  text.  After  reading,  the  text  was  removed  and  the  questions 
were  again  presented.  Results  were  the  same  as  in  the  first  study,  easier 
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versions  did  not  produce  improved  comprehension.  In  Study  3  the  passages 
were  presented  in  the  form  of  cloze  tests  with  no  time  limits.  Again, 
easier  versions  did  not  produce  improved  comprehension.  And  finally, 
testing  conditions  in  Study  4  paralleled  the  "reading  to  learn"  conditions 
of  a  classroom  setting.  Sailors  were  allowed  to  study  the  passages  with 
no  time  limits  imposed.  Passages  were  then  taken  away  and  the  test  was 
administered.  Once  again  easier  passages  did  not  produce  Improved 
comprehension  for  the  high  and  mid-level  reading  skill  groups.  The  low 
reading  skill  group  (7  to  8.5  RGL)  did  show  an  increase  in  comprehension. 
However,  the  greatest  Increase  (38  percent  on  original  version  vs  53 
percent)  was  on  the  set  of  passages  which  had  an  average  readability 
score  at  the  10th  grade  level  and  was  the  most  difficult  of  the  easier 
versions . 

Conclusion:  Rewriting  To  Lower  The  Formula  Score  Does  Not  Increase 
Comprehension.  Civilian  and  military  research  reviewed  above  suggests 
that  lowering  difficulty  of  text  by  rewriting  to  formula  does  not  enable 
readers  to  comprehend  material  they  could  not  comprehend  from  the  more 
difficult  version. 

How  Will  Formula  Score  Requirements  Affect  Production  Of  Written  Material? 

The  Army's  major  purpose  in  establishing  Reading  Grade  Level  (RGL) 
standards  for  material  is  to  insure  a  closer  match  between  difficulty 
level  of  the  material  and  reading  skill  level  of  the  intended  readers. 

It  has  been  assumed  that  the  RGL  scores  obtained  from  readability  formulas 
identify  the  reading  skill  level  (also  expressed  in  RGLs)  at  which 
readers  will  adequately  comprehend  the  material.  Research  reviewed 
above  clearly  indicates  that  this  is  a  false  assumption.  Readability 
formulas  do  not  possess  the  predictive  power  to  identify  reading  skill 
levels  at  which  specific  material  will  be  adequately  comprehended.  Data 
reviewed  suggests  that  their  predictive  power  is  limited  to  predicting 
that  material  written  above  the  10th  RGL  will  generally  be  too  difficult 
for  soldiers  whose  reading  skills  fall  below  the  10th  RGL.  Other 
research  reviewed  above  indicates  that  rewriting  "difficult"  text  to 
"easier"  readability  score  levels  has  no  practical  effect  on  improving 
readers'  comprehension  of  the  text.  Thus,  use  of  readability  formula 
scores  to  determine  when  reading  skill  level  standards  have  been  met  and 
when  material  must  be  rewritten  to  satisfy  the  standards  will  be  Ineffective 
in  achieving  Army  objectives. 

In  addition  to  being  inaccurate,  use  of  formula  scores  to  identify 
readability  standards  has  other  serious  drawbacks.  Formulas  do  not 
require  the  writer  and  editor  to  examine  the  writing  for  clarity  in 
addressing  the  targeted  readers'  information  needs.  When  imposed  as  a 
standard  that  must  be  met,  attention  tends  to  be  focused  on  meeting  the 
score  requirement  rather  than  on  organizing  the  material  to  meet  the 
readers'  information  needs. 
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Accommodations  made  to  enable  writers  to  meet  formula  standards  may 
even  promote  the  production  of  less  readable  writing.  For  example,  the 
writer  is  often  caught  in  the  crunch  of  preparing  material  in  a  job  area 
that  has  a  large,  well  established  technical  vocabulary.  These  are 
usually  multi-syllable  words.  It  is  generally  considered  mandatory  that 
the  writer  use  these  words.  The  Army  Materiel  Command  (now  Army  Materiel 
Development  &  Readiness  Command)  estimated  in  1970  that  approximately  one- 
third  of  the  words  used  in  their  technical  manuals  were  mandatory  words 
(Army  Materiel  Command,  1970).  This  proportion  of  multi-syllable  words 
alone  would  result  in  a  FORCAST  estimate  of  reading  difficulty  at  the 
10th  grade  level.  This  poses  a  problem  for  the  writer  who  must  use 
these  words  yet  still  stay  within  formula  score  requirements.  A  common 
solution  to  this  probelm  is  to  keep  the  same  formula  ..core  requirement 
but  adjust  either  the  formula  or  the.  method  of  counting.  The  effect  of 
these  adjustments  is  to  pretend  the  technical  terms  aren't  there  for 
purposes  of  computing  the  score.  This  solution  is  justified  by  the 
argument  that  technical  words  either  are  or  will  quickly  become  familiar 
words  to  everyone,  from  entry  level  on  up,  working  in  the  technical 
area.  Apparently,  efforts  to  verify  this  assumption  are  rarely  made. 

Those  that  have  been  reported  fail  to  support  it  (Jablonski,  1971;  Hooke 
et  al.,  1979). 

Common  acceptance  of  the  assumption  that  technical  terms  are  well 
known  terms  illustrates  how  the  push  to  meet  formula  score  standards  can 
actually  promote  production  of  less  readable  writing.  Given  this 
assumption  of  word  familiarity,  writers  do  not  have  to  focus  their 
attention  on  how  to  best  communicate  technical  information  to  readers 
who  have  limited  understanding  of  the  technical  area  and  its  jargon. 

The  Air  Force  recently  completed  a  research  study  of  the  effectiveness 
of  their  readability  standards  in  improving  comprehensibility  of  publications 
in  seven  career  fields.  A  major  implication  for  the  Air  Force  cited  in 
this  study  is  as  follows: 

The  policy  of  writing  to  target  audience  should  be  continued  but 
should  not  be  enforced  rigidly.  Given  the  relatively  crude  ways 
employed  at  present  to  estimate  literacy  gap,  it  is  probably  not 
appropriate  to  insist  that  writers  hit  their  targets  with  a  great 
deal  of  precision.  Additionally,  the  practical  problems  involved 
in  simplifying  materials  below  tenth  grade  level  may  be  insur¬ 
mountable  (Hooke  et  al.,  1979). 

Conclusion:  Requiring  that  text  be  written  to  satisfy  a  targeted 
reading  grade  level  score  shifts  attention  to  meeting  the  score  requirement 
rather  than  focusing  attention  on  organizing  material  to  meet  the  readers' 
information  needs. 
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A  MORE  EFFECTIVE  APPROACH  TO  ARMY  OBJECTIVES 


The  recent  focus  on  readability  formulas  has  diverted  attention  from 
developing  and  improving  other  approaches  to  readability  standards.  As 
stated  at  the  beginning  of  this  paper,  to  achieve  Army  readability 
objectives,  publications  must  satisfy  two  interrelated  conditions.  They 
must:  (a)  clearly  address  the  job  and  training  needs  of  the  target 
audience,  and  (b)  be  written  in  a  style  that  is  easily  comprehended  by 
soldiers  in  the  target  audience.  A  readability  formula  provides  no 
information  on  how  well  writing  meets  the  job  and  training  needs  of  the 
target  audience.  Formula  scores  provide  only  a  gross  index  to  how  well 
the  writer  has  achieved  an  easily  comprehended  style.  As  numerous 
supporters  and  critics  of  readability  formulas  have  pointed  out  the 
writing  could  be  completely  nonsensical  and  still  obtain  an  "easily 
readable"  formula  score.  Thus,  readability  formulas  can  serve  at  best, 
only  a  minor  role  in  achieving  Army  objectives  for  improved  readability 
of  publications.  Achievement  of  Army  objectives  requires  use  of  a 
broader-based  method  for  establishing  readability  standards  and  for 
evaluating  written  material  against  these  standards. 

A  broader-based  method,  appropriate  for  the  Army's  use,  does  not 
exist  and  will  have  to  be  developed.  However,  general  models  for  such 
a  method  and  many  of  the  components  that  would  be  used  in  developing  the 
method  do  exist.  The  basic  model  is  the  checklist  used  by  editors  and 
teachers  of  technical  writing  to  identify  major  features  of  a  manuscript 
which,  by  their  judgment,  are  important  contributors  to  its  final  readability. 
Content  and  level  of  detail  in  existing  checklists  depends  on  the  developer's 
special  Interests.  Checklists  ir  technical  writing  texts  tend  to  focus 
on  general  principles  of  content  organization  rather  than  specific  types 
of  material.  The  usual  editor's  checklist  tends  to  focus  primarily  on 
form  (grammar,  punctuation,  spelling  and  format)  rather  than  on  function 
(organization  to  convey  information,  depth  and  scope  of  coverage,  and  so 
forth).  Checklists  developed  to  accompany  Military  Specifications 
become  extremely  long  and  detailed.  Figures  1  and  2  present  the  table 
of  contents  for  a  checklist  prepared  for  use  by  the  Navy  in  conducting 
quality  assurance  (Price,  1975).  Figure  3  shows  the  major  headings  of 
a  much  shorter  checklist  developed  by  Air  Force  researchers  to  rate 
quality  of  manuals  based  on  the  presence  or  absence  of  problems  users  in 
the  field  encountered  in  trying  to  use  the  manuals  (Ross,  1959).  None 
of  these  checklists  are  directly  appropriate  for  present  Army  objectives. 

The  point  is  that  checklists  can  be  designed  to  identify  major  types  of 
problems  judged  to  make  reader  comprehension  of  the  message  more  difficult. 

They  can  be  tailored  to  the  special  kinds  of  problems  Army  writers 
encounter  in  communicating  different  kinds  of  material  (for  example, 
administrative  regulations  versus  job  training  materials).  Their  effectiveness 
can  even  be  submitted  to  experimental  validation.  And  finally,  they  can 
provide  more  useful  diagnostic  guidance  to  the  writer  on  how  to  go  about 
a  rewrite  than  is  provided  by  readability  formulas. 
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(Price,  1975) 
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I.  Layout 
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Figure  3.  Major  Headings  From: 

CHECKLIST  FOR  EVALUATION  OF  TECHNICAL  MANUALS 
(Ross,  1959) 
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Development  of  an  Army  Readability  Standards  Checklist  requires  a 
research  and  development  effort.  This  checklist  should  be  developed  to 
meet  the  following  objectives:  (a)  Identifies  specific  problems  that 
need  to  be  corrected  when  standards  are  not  met;  (b)  is  not  limited  to 
only  connected  prose  but  Is  applicable  to  the  entire  manuscript  (for 
example.  Illustrations,  tables  and  Index);  and,  (c)  does  not  require 
extensive  technical  editing  and  writing  expertise  to  apply.  Extent  of 
this  effort  will  depend  on  the  final  objectives  established  for  the 
checklist.  Initial  development  will  focus  on  working  with  Army  writers 
and  editors  to  Identify  content  for  the  different  major  types  of  literature 
produced.  Draft  checklists  will  be  tried  out  by  writers  and  editors  at 
different  levels  in  the  production  chain.  Two  major  criteria  will  be 
employed  during  this  developmental  process :  (a)  extent  of  agreement 

among  writers  and  editors  In  making  specific  judgments  on  the  same 
material,  and  (b)  relationship  of  these  judgments  to  users'  judgments  of 
comprehensibility  of  the  material. 

Development  of  the  type  of  checklist  described  above  will  require 
analyses  of  the  different  types  of  material  currently  being  produced. 

An  example  of  ;his  type  of  analysis  and  the  kinds  of  problems  writers 
and  editors  might  expect  to  find  Is  provided  by  research  conducted  for 
Army  Research  Institute  (ARI).  This  research  was  initiated  In  the  early 
1970's  In  response  to  the  Army's  concern  over  the  high  reading  difficulty 
levels  of  training  and  job  materials  entry  level  enlisted  personnel  were 
expected  to  use.  Part  of  this  research  involved  performing  an  extensive 
analysis  of  the  contents  of  Army  Field  Manuals  and  Technical  Manuals. 

This  analysis  resulted  in  the  identification  of  a  number  of  writing 
problems,  summarized  under  seven  major  problem  areas  (Kern  et  al., 

1977): 

(a)  Hard  words,  long  sentences. 

(b)  Topic  oriented  writing  -  not  directed  to  any  particular  Job 
user. 

(c)  Main  points  lost  In  detail  -  too  vague,  too  wordy. 

(d)  Confused  development  -  main  points  fragmented  across  paragraphs. 

(e)  Remote  references  -  text  relies  on  material  located  elsewhere. 

(f)  Weak  visual  aids  -  absent,  illegible,  not  related  to  text. 

(g)  Run  together  format  -  lengthy,  solid  masses  of  print. 
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Pages  from  Field  Manuals  (FMs)  and  Technical  Manuals  (TMs)  were  selected 
to  illustrate  these  problems.  The  final  sample  of  pages  reflected 
technical,  combat  and  administrative  content  produced  by  writers  from  11 
different  Army  schools.  Each  sample  was  rewritten  to  provide  a  clearer, 
less  difficult  version  of  the  material.  The  rewriting,  performed  by 
research  staff  members,  focused  on  the  seven  problem  areas  listed  above. 
Note  that  only  one  of  the  seven  problem  areas  listed,  the  first  one,  can 
be  addressed  by  writing  to  a  readability  formula.  Preparation  of  clearer, 
less  difficult  versions  of  the  pages  sampled  from  FMs  and  TMs  required 
extensive  structuring  and  reorganization  of  the  way  the  information  was 
presented.  These  changes  are  illustrated  visually  by  the  "Before"  and 
"After"  examples  presented  in  the  Guidebook  (Kern  et  al.,  1977). 

As  demonstrated  earlier,  writing  to  a  formula  does  not  improve 
readers  comprehension  of  the  material.  The  criterion  for  readability 
should  be  one  that  writers  and  editors  could  be  encouraged  to  use  as  a 
guide  to  improve  their  writing.  Research  conducted  in  1974  for  ARI 
indicated  that  most  Army  writers  are  instructors  who  have  not  had  formal 
training  in  technical  writing  (Appendix) .  Editors  tended  to  confine 
their  review  of  manuscripts  to  punctuation,  spelling  and  format  regulations. 
A  checklist  which  Identifies  common  problems  in  organizing  and  presenting 
information  to  meet  the  readers  needs  could  also  assist  the  editors  in 
performing  their  Job  more  effectively. 
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APPENDIX 


The  Study  of  Writing  in  Functional  Contexts:  The  Army  Writer 
and  the  Design  of  a  Guidebook  for  Developing 
Army  Training  Literature 


Richard  P.  Kern  and  Thomas  G.  Sticht 
Human  Resources  Research  Organization  (HumRRO) 
March  1974 
Prepared  For: 

U.  S.  Army  Research  Institute 
Contract:  S73-33  (ATL) ,  DAHC19-73-C-0051 


This  is  a  special  report  designed  for  management  on  information 
developed  in  HumRRO  Work  Unit  ATL  (Army  Training  Literature) , 

Tasks  4,  5,  and  6  (Development  of  a  Guidebook  for  Writers  of  Army 
Training  Literature) .  The  report  summarizes  interviews  with  Army 
writers.  It  presents  the  implications  of  this  information  for 
the  design  of  the  guidebook  and  for  policy  regarding  Army  training 
literature  development. 


« 


PAQS.  BLANK  -  NOT  fllM SD 


GUIDEBOOK  FOR  THE  DEVELOPMENT  OF  ARMY  TRAINING  LITERATURE: 

RATIONALE  AND  POLICY  IMPLICATIONS  FROM  INTERVIEWS  WITH  ARMY  WRITERS 

Contract:  S73-33  (ATL) 

DAHC19-73-C-0051 

PROJECT  REQUIREMENTS:  To  design  a  Guidebook  for  Army  writers  which  will 

actually  be  used,  information  is  needed  about  the 
Army  writer  and  his  job. 

PROCEDURE:  Conduct  interviews  with  Army  writers  to  find  out  who 

they  are,  how  they  do  their  jobs,  and  what  they  think 
a  writer's  guidebook  should  include  to  be  of  help 
to  them. 

INTERVIEW  POPULATION:  Army  writers  and  editors  located  at  five  Army  Service 

School  Centers  chosen  to  reflect  agencies  responsible 
for  producing  literature  supporting  relatively  high 
density  Military  Occupational  Specialty  (MOS)  areas. 


School 

MOS  Area  No. 

Interviewed 

Infantry  School,  Fort  Benning 

Combat 

19 

U.S.  Army  Signal  Center  and 

Technical 

24 

School,  Fort  Monmouth 

U.S.  Army  Ordnance  Center  and 

Technical 

25 

School,  Aberdeen  Proving 
Ground 

U.S.  Army  Quartermaster 

Technical 

9 

School,  Fort  Lee 

U.S.  Army  Institute  of 

Administrative 

17 

Administration,  Fort 

Benjamin  Harrison 

Total 

94 

MAJOR  CONCERNS: 

I.  The  Army  Writer 

Who  are  Army  writers? 

‘  What  are  their  qualifications? 

J  How  is  their  time  distributed  over  writing  and 
instruction? 

II.  Use  of  Writers'  Guidebooks 

Do  writers  use  guidebooks? 

What  information  would  writers  find  useful  in  a 
guidebook? 

III.  The  Current  Writing  Process 

What  information  is  the  writer  given  in  a  writing 
assignment? 

What  types  of  support  does  the  writer  have  during 
the  writing  process? 

What  type  of  quality  control  is  used  prior  to 
publication? 

How  is  feedback  obtained  from  users? 

IV.  Nature  of  Army  Training  Literature  (ATL) 

How  are  Army  Training  needs  identified? 

Is  ATL  topic  or  job  performance  oriented? 

Are  training  and  reference  functions  of  ATL 
distingu ished? 
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CONCERN  1: 


THE  ARMY  WRITER 


QUESTIONS:  WHO  ARE  ARMY  WRITERS? 

WHAT  ARE  THEIR  QUALIFICATIONS? 

SUMMARY  OF  INTERVIEWS 


Writers  of  Army«Wide  Literature  are  usually  personnel  who  are  consi¬ 
dered  subject  matter  experts  in  the  assigned  writing  area,  and  who 
are  instructors  in  the  area.  At  the  Infantry  School,  writers  were 
predominantly  military  officers.  At  the  technical  and  administra¬ 
tive  schools  writers  were  predominantly  civilian  personnel  and 
either  present  or  former  instructors.  Qualifications  for  military 
and  civilian  writers  were  usually  stated  as:  (a)  subject  matter  ex¬ 
perts;  (b)  experienced  as  instructors;  (c)  demonstrated  ability  to 
write  instructional  materials.  Most  writers  were  college  graduates 
with  no  formal  training  in  technical  writing. 


QUESTION:  HOW  IS  THEIR  TIME  DISTRIBUTED  OVER  WRITING 

AND  INSTRUCTION? 


The  writing  assignment  is  an  extra  duty  for  most  writers  who  must 
also  instruct.  Writers  pointed  out  that  in  the  school  setting,  first 
priority  is  given  to  instruction  and  the  operational  requirements 
associated  with  supporting  instruction.  Writing  manuals  is  a  low 
priority  effort  given  little  support.  The  instructor-writer  felt 
conflicts  between  the  time  required  to  perform  his  instructional 
duties  and  the  time  required  to  research  and  write  an  adequate  man¬ 
ual,  with  manual  writing  suffering  from  lack  of  time.  Among  mili¬ 
tary  writers  a  high  rate  of  turnover  exists,  with  writing  assign¬ 
ments  being  passed  on  from  man-to-man. 


IMPLICATIONS 


For  The  Guidebook:  The  part-time  nature  of  the  writing  job  suggests 
that  the  Guidebook  must  be  designed  to  be -used  by  people  who  cannot 
and  will  not  devote  considerable  time  to  study  to  learn  to  be  better 
writers.  Hence  the  Guidebook  should  avoid  trying  to  train  writers, 
and  instead  minimize  the  amount  of  "mental  energy"  the  writers  must 
expend  to  use  the  Guidebook.  As  much  as  possible  the  attempt  should 
be  to: 

*  Catalog  information  in  the  Guidebook,  not  in  the  writer's  head. 

*  Provide  aids  for  decision  making  and  accomplishing  recurring 

procedures . 

*  Emphasize  intuitive  understanding  by  examples  rather  than 

lectures  on  "how  to  do  it." 

For  Army  Policy:  The  critical  position  of  Army  Training  Literature 
writers  needs  considerably  more  identity  and  stability  if  writers 
are  to  effectively  produce  and  also  develop  skills  and  techniques  to 
meet  the  changing  needs  of  the  Army.  The  Army  should  consider 
creating  permanent,  full-time  technical  writers  slots  for  the  pro¬ 
duction  of  Army  Wide  training  manuals.  Then  it  would  not  be  un¬ 
reasonable  to  expect  these  personnel  to  undergo  formal  training  in 
effective  writing  and  to  continue  to  update  their  knowledge  of  ways 
to  use  printed  communication  to  support  training  and  job  perform¬ 
ance. 


CONCERN  II: 


USE  OF  WRITERS'  GUIDEBOOKS 


QUESTIONS:  DO  WRITERS  USE  GUIDEBOOKS? 

WHAT  INFORMATION  WOULD  WRITERS  FIND  USEFUL 
IN  A  GUIDEBOOK? 


SUMMARY  OF  INTERVIEWS 


Most  of  the  writers  interviewed  said  they  do  not  use  writer's  guides 
of  any  type.  They  get  the  guidance  they  need  regarding  format  and 
organization  by  reviewing  previous  editions  of  the  same  manual  or  a 
related  manual.  Some  writers,  however,  felt  the  following  addition¬ 
al  types  of  guidance  would  be  helpful: 

a.  identify  reference  sources  for  literature  research 

b.  how  to  conduct  literature  research 

c.  outline  steps  necessary  for  writing,  going  from 

inception  to  final  review 

d.  how  to  develop  outlines 

e.  how  to  organize  material  to  facilitate  learning 

f.  how  to  determine  reading  grade  level  of  material 

g.  how  to  write  at  predetermined  grade  level 

h.  when  is  it  appropriate  to  be  redundant? 

i.  when  is  it  important  to  summarize? 

j .  guidance  in  types  and  best  use  of  illustrations 

k.  how  to  obtain  copyright  release 

l.  guidance  in  determining  user,  purpose,  and  scope 


CONCERN  III:  THE  CURRENT  WRITING  PROCESS 

QUESTION:  WHAT  INFORMATION  IS  A  WRITER  GIVEN  IN  A 

WRITING  ASSIGNMENT? 


Writers  appear  to  be  provided  with  little  or  no  guidance  regarding 
users,  purpose  or  scope  when  a  new  manual  is  to  be  written.  Because 
the  writer  is  a  subject  matter  expert,  he  is  expected  to  define 
these  objectives.  Lack  of  consistency  in  identifying  users,  pur¬ 
pose  and  scope  was  evidenced.  Users,  when  identified,  are  usually 
identified  in  terms  of  the  organization  (e.g.  ,  Company  or  Battalion) 
or  the  Army  at  large  (e.g.,  everyone  from  Private  to  General)  and 
rarely  in  terms  of  specific  job  positions  within  any  organization. 
For  revisions,  writers  generally  never  see  files  containing  the 
previous  manuscript,  review  comments,  and  change  recommendations 
that  have  come  in  since  its  publication. 
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IMPLICATIONS 


For  The  Guidebook:  Many  writer's  guides  are  available  in  the  Army 
today  which  provide  extensive  discussions  of  many  of  the  topics  men¬ 
tioned  by  writers  as  candidates  for  a  writer's  guide,  yet  writers 
do  not  use  these  guides.  As  suggested  above,  the  ATL  Guidebook 
should  try  to  capitalize  on  the  fact  that  writers  get  ideas  about 
format  and  organization  by  reviewing  a  previous  edition  of  a  manual 
or  related  manuals.  Thus,  the  ATL  Guidebook  should  provide  a 
"sampler"  of  various  formats  in  different  content  areas  which  writers 
can  get  ideas  from  for  use  in  their  manuals.  Guidance  should  be 
given  for  identifying  users,  for  determining  writing  objectives,  and 
for  selecting,  assembling  and  organizing  content. 

For  Army  Policy :  A  concerted  effort  should  be  made  to  make  current 
Army  writer's  guides  available  to  all  writers.  Continued  education 
of  the  writers  regarding  the  contents  of  various  guides  is  needed. 
This  goes  hand-in-hand  with  the  recognition  of  the  writing  job  as 
a  professional  position  within  the  ATL  system. 


For  Amy  Policy:  The  writer's  writing  assignment  should  specify  the 
purpose  of  the  manual,  the  intended  major  user,  the  subject  area, 
and  what  duties  and  tasks  the  writer  is  expected  to  relate  the  sub¬ 
ject  area  to.  For  revisions,  the  writer  should  also  be  given  a  file 
containing  all  comments  and  change  recommendations  that  have  come  in 
since  the  last  publication  of  the  manual. 
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CONCERN  III:  (CONT.) 

QUESTIONS:  WHAT  TYPES  OF  SUPPORT  DOES  THE  WRITER  HAVE 

DURING  THE  WRITING  PROCESS? 

WHAT  TYPE  OF  QUALITY  CONTROL  IS  USED  PRIOR 
TO  PUBLICATIONS. 


SUMMARY  OF  INTERVIEWS 


To  research  information  writers  rely  heavily  on  school  lesson  plan 
files  and  previous  editions  of  the  manual.  Whenever  possible  they 
"cut-and-paste"  from  the  earlier  manuals  as  well  as  from  current 
editions  of  related  manuals.  They  generally  are  dependent  upon  the 
routine  Post  training  aid  support  for  illustrators;  contact  with  the 
illustrator  tends  to  be  of  a  paper-work,  through  channels  nature  and 
not  face-to-face  communication.  This,  plus  the  experience  of  long 
time  delays  cause  most  writers  to  avoid  any  use  of  illustrations 
beyond  their  own  capabilities  or  beyond  those  that  can  be  cut-and- 
pasted  from  other  manuals.  Writers  did  not  use  readability  formu¬ 
las  or  other  methods  for  testing  the  difficulty  of  their  writing. 
Editorial  review  of  the  writer's  draft  is  generally  limited  to 
grammar,  proof  reading  and  conformance  to  format  policies.  There  is 
apparently  no  review  by  the  writer  or  editor  of  copy  set-up  by  the 
printer  prior  to  production  printing. 


QUESTION:  HOW  IS  FEEDBACK  OBTAINED  FROM  USERS? 


Generally,  the  only  feedback  writers  receive  from  users  is  from  the 
students  in  the  classroom.  Field  reviews  were  considered  ineffec¬ 
tive  formalities.  Suspense  dates  for  review  were  considered  too 
immediate  to  allow  adequate  review.  Writers  who  had  served  as  re¬ 
viewers  stated  that  once  their  written  comments  were  provided,  they 
learned  nothing  of  the  acceptance  or  rejection  of  their  comments. 
Instructors  also  considered  the  filling  out  of  recommended  change 
forms  (Form  2028)  as  futile  because  they  saw  the  same  errors  repeat¬ 
ed  in  future  revisions.  Generally,  writers  reported  they  had  never 
seen  Field  Review  comments  on  manuals  they  had  written  and  had  no 
idea  whose  comments  had  even  been  solicited. 
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IMPLICATIONS 


For  The  Guidebook:  Writers  need  information  about  the  users  of  their 
products.  The  Guidebook  should  give  guidance  in: 

*  identifying  primary  and  other  users 

*  determining  reading  abilities  of  users 

The  guidebook  should  give  guidance  in  using  information  about  users 
to: 

*  define  purpose  and  scope  of  manuals  in  terms  of  users 

*  write  at  the  reading  levels  of  the  primary  users 

The  guidebook  should  acknowledge  the  writer's  tendency  to  imitate 
past  work,  especially  under  time  pressures,  and  provide  a  ready 
source  of  a  large  set  of  samples  of  bad  and  good  writing,  formatting, 
and  illustrating  which  writers  could  examine  and  imitate  the  good 
samples.  Guidance  should  be  provided  for  the  use  of  the  FORCAST 
readability  formula  developed  for  use  with  Army  manuals  as  a  quality- 
control  tool  during  writing  and  review  of  drafts. 

For  Army  Policy:  Again,  the  part-time  nature  of  the  writing  posi¬ 
tion  places  undue  pressures  on  time.  Higher  priority  to  the  writing 
of  ATL,  including  a  better  defined  and  more  permanent  position, 
would  appear  desirable.  Research  reference  files  should  be  maintained 
and  made  available  to  the  writers  rather  than  being  an  individual 
writer's  responsibility.  These  files  should  also  include  available 
task  analyses  information  on  the  various  users  to  whom  manuals  may  be 
addressed.  Provisions  should  be  made  for  more  direct  illustrator 
support  with  the  illustrator  a  member  of  the  writing  staff.  Quality 
control  during  writing  and  processing  of  the  draft  should  be  reviewed 
for  approaches  to  improving  it. 


For  The  Guidebook:  Since  the  only  feedback  writers  receive  is  from 
students  in  the  classroom  setting,  the  Guidebook  should  stress  the 
use  of  formal  or  informal  task  analysis  information  to  help  the  writer 
in  addressing  their  manuals  to  user's  needs  in  the  field.  Contacts 
limited  to  school  students  also  mean  that  writers  have  little  or  no 
contact  with  users  in  the  field  who  are  less  capable  readers  than 
they  or  their  students.  Therefore,  the  Guidebook  should  help  the 
writer  understand  the  nature  of  the  problems  the  poorer  reader  faces 
in  extracting  information  from  printed  materials. 

For  Army  Policy:  The  current  system  does  not  provide  feedback  that 
writers  or  literature  planners  can  use  to  tell  whether  a  manual, 
once  produced,  meets  the  needs  of  the  intended  user,  or,  in  fact,  is 
even  used.  The  Army  should  consider  a  new  system  to  either  supple¬ 
ment  or  replace  the  Field  Review  process.  The  new  system  should  be 
a  regular  system  managed  by  a  central  authority  (e.g. ,  TRADOC) .  It 
might,  for  example,  collect  information  from  users  in  the  field  on 
the  utility  and  acceptability  (readability  and  readership)  of  train¬ 
ing  literature  publications  and  feed  this  information  back  to  the 
training  literature  branch  which  prepared  the  publications.  If 
this  information  were  collected  at  the  direct  user  level  and  not 
just  at  the  field  command  level  it  should  provide  useful  guidance 
for  both  the  planning  and  writing  of  ATL. 
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CONCERN  IV: 
QUESTION: 


NATURE  OF  ARMY  TRAINING  LITERATURE 

HOW  ARE  ARMY  TRAINING  LITERATURE  NEEDS 
IDENTIFIED? 


SUMMARY  OF  INTERVIEWS 


Under  the  present  system  school  staff  carry  a  major  responsibility 
for  identifying  Army  Wide  literature  needs.  Each  school  periodi¬ 
cally  reviews  the  Field  Manuals,  Technical  Manuals,  etc.,  for  which 
they  are  responsible.  From  this  review  recommendations  are  made 
for  updating  or  preparing  new  material.  The  school  consolidates 
these  recommendations,  including  any  comments  received  from  the  field, 
and  staffs  them  through  TRADOC  for  approval,  budgeting  and  priori¬ 
ties.  Manuals  are  then  assigned  to  the  appropriate  branch  or  instruc¬ 
tional  department,  where  specific  manuals  are,  in  turn,  assigned  to 
the  person  who  will  do  the  actual  writing.  When  the  assignment 
reaches  the  writer  it  is  frequently  simply  a  directive  to  write  a 
manual  on  "subject  X". 


QUESTIONS:  IS  ATL  TOPIC  OR  JOB  PERFORMANCE  ORIENTED? 

ARE  TRAINING  AND  REFERENCE  FUNCTIONS  OF 
ATL  DISTINGUISHED? 


Writers  view  their  jobs  as  that  of  assembling,  organizing  and  record¬ 
ing  what  is  known  about  a  topic.  It  is  assumed  that  if  a  manual 
contains  a  reasonably  complete  exposition  of  the  topic,  it  can  serve 
as  a  general  text  or  reference  source  and  be  used  by  anyone  for  any 
purpose,  whether  initial  learning,  looking-up  previously  learned 
information,  or  consulting  for  directing  the  training,  learning  and 
job  performance  of  others.  Manuals  were  only  rarely  considered  in 
terms  of  job  duties  or  task  performance. 
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IMPLICATIONS 


For  Army  Policy:  A  clear  identification  of  the  purpose  and  intended 
major  user  of  AIL  would  r.eem  to  be  basic  to  planning  literature  needs 
and  priorities  for  the  ATL  Program.  This  information  should  then 
become  part  of  the  writer's  assignment  rather  than  charging  the  writer 
with  determining  purpose  and  user  after  the  decision  has  been  made 
to  write  the  manual. 


For  The  Guidebook:  Guidance  should  be  provided  on  how  to  prepare 
literature  oriented  to  job  performance  of  users,  including  design  of 
manuals  for  more  than  one  major  user  (e.g. ,  commanders ,  supervisors, 
workers,  instructors).  Guidance  should  be  provided  on  the  design  of 
manuals  for  flexibility  of  purpose  (e.g.,  for  initial  learning  and 
for  later  reference). 

For  Army  Policy :  The  same  manuals  designed  for  presentation  of 
general,  topic-oriented  information  are  not  effective  as  instruc¬ 
tional  texts  for  new  learners,  as  job  aids  for  directing  Job  per¬ 
formance,  and  as  reference  sources  for  experienced  workers. 

Emplusis  should  be  given  to  policy  which  recognizes  the  different 
functions  of  ATL  and  the  design  implications  of  these  functions. 
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1  Naval  Aerospc  Med  Res  Lab,  Pensacol*,  ATTN:  Cod*  LSI 
1  N*v»l  Aerospc  Med  Res  Lab,  Pensacola.  ATTN:  Code  L5 
1  Chiel  of  NevPen.  ATTN:  PersOR 
I  NAVAIflSTA.  Norfolk.  ATTN:  Safety  Ctr 
I  Nev  Oceanogrepbic.  DC.  ATTN:  Code  6251,  Charts  &  Tach 
I  Center  of  Nevel  Anal,  ATTN:  Doc  Ctr 
I  NavAirSysCom,  ATTN:  AIR— S313C 
1  Nav  BuMed.  ATTN:  1 13 
I  NavHelicopterSubSqua  2.  FPO  SF  90401 
1  AFHRL  (FT)  Williams  AF8 

1  AFHflL  (TT)  Lowry  AFB 
t  AFHRUAS1  WPAFB.OH 

2  AFHRL  IOOJZ)  Brooks  AFB 

1  AFHRL  IDOJN1  Lackland  AFB 
1  HQUSAF  (INYSDI 
1  HQUSAF I0PXXA) 

1  AFVTG  (ROl  Randolph  AFB 

3  AMRL  (HEI  WPAFB.OH 

2  Af  Inst  ol  Tech.  WPAFB.  OH.  ATTN:  ENErSL 
1  ATC  (XPTOI  Randolph  AFB 

1  USAF  AeroMed  Lib.  Brooks  AFB  (SUL-41,  ATTN:  OOC  SEC 
1  AFOSR  INU,  Arlington 

1  AF  Log  Cmd.  McClellan  AFB.  ATTN:  ALC/DPCRB 

1  Air  Force  Academy.  CO.  ATTN:  D*P<  of  8*1  Sen 
5  Nev  Pert  &  Oev  Ctr.  San  Diego 

2  Nevy  Med  Neuropsychietric  Rscb  Unit.  San  Diego 
t  Nev  Electronic  Lab.  Sen  Diego.  ATTN:  Res  Lab 

1  Nev  TrngCen.  San  Oiego.  ATTN:  Code  9000- Lib 
1  NevPostGreSch,  Monterey.  ATTN:  Code  55Aa 
I  NevPostGreSch,  Monterey.  ATTN:  Code  2124 
I  NevTrngEquipCtr,  Orlando,  ATTN:  Tech  Lib 
1  US  Oept  of  Labor,  DC,  ATTN:  Manpower  Admin 
1  US  Oept  of  Justice,  DC.  ATTN:  Drug  Enforce  Admin 
1  Nat  Bur  of  Standards,  DC,  ATTN:  Computer  Info  Section 
1  Nat  Clearing  House  for  MH-Info.  Rockville 
I  Denver  Federal  Ctr.  Lakewood.  ATTN:  8LM 
12  Defense  Documentation  Center 
4  Dir  Psvch,  Army  Hq,  Ruseell  Ofcs.  Canberra 
1  Scientific  Advtr,  Mil  8d.  Army  Hq,  Russell  Ofcs,  Canberra 

1  Mil  and  Air  Attach*.  Austrian  Embassy 

(  C«mr«  d*  R*ch*rch«  0«s  Ftcteurt,  Humain*  da  la  Defense 
Netionsle.  Brussels 

2  Canedien  Joint  Staff  Washington 

t  C’Air  Staff,  Royal  Canadian  AF,  ATTN:  Pars  Std  Anal  Br 

3  Chief,  Canadian  Def  Rsch  Staff.  ATTN:  C,'CRDS(W1 

4  British  Oef  Staff.  8ntish  Embassy,  Washington 


1  Oef  Si  Civil  Inst  of  Enviro  Medicine,  Canada 
I  AIR  CRESS  k.erinryiton,  ATTN:  Info  S'.'t  Br 
1  Militaerpsykologisk  rjeneate.  Cooanhagen 
1  Military  Attache.  French  Embaasv.  ATTN:  Doc  S' 
I  Medectn  Chef.  C.E.R.P.A.- Arsenal.  Toulon/Nevr 
1  Prm  Scientific  Qlt.  Appl  Hum  Engr  Rich  Oiv,  Mir 
of  Defense.  New  Delhi 

1  Pers  Rsch  Ofc  Library.  AKA,  Israel  Defense  Foret 
I  Ministers  van  Defensie,  DQOP/KL  Afd  Sociaal 
Psychologiscbe  Zaken.  The  Hague.  Netherlands 


